<html>
<head><meta charset="utf-8"><title>playground metrics tracking · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html">playground metrics tracking</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="236831219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236831219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236831219">(Apr 30 2021 at 13:09)</a>:</h4>
<p>I'm looking at adding some basic prometheus metrics to the playground to at least get the scope of the recent "timeouts when building" issue. The most basic thing I can think of is something like</p>
<div class="codehilite"><pre><span></span><code>requests { endpoint: &quot;execute|compile|format...&quot;, outcome: &quot;success|soft-timeout|hard-timeout|...&quot; }
</code></pre></div>
<p>However, there's a lot of richer information available on a per endpoint basis. For example, many endpoints have a <code>channel: "stable|beta|nightly"</code> and some have very specific things like <code>asm_flavor: "intel|att"</code>. Based on my one hour of working, it appears that you can't have "missing" labels on a metric, so I'm thinking of having something like</p>
<div class="codehilite"><pre><span></span><code>requests { endpoint, outcome }
execute_requests { outcome, channel, ... }
format_requests { outcome, ... }
</code></pre></div>
<p>Does that metrics design make sense?</p>
<p>/cc <span class="user-mention" data-user-id="121055">@Pietro Albini</span> <span class="user-mention" data-user-id="232545">@Joshua Nelson</span> (<span class="user-mention" data-user-id="116122">@simulacrum</span> said you might have some useful experience in the general metrics area)</p>



<a name="236831374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236831374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236831374">(Apr 30 2021 at 13:10)</a>:</h4>
<p>My worry is that an execute request will result in two metrics being incremented, kind of double-counting. I'm not sure if that's a problem or not.</p>



<a name="236832135"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236832135" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Joshua Nelson <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236832135">(Apr 30 2021 at 13:16)</a>:</h4>
<p>that seems reasonable, there's no problem with two different metrics as long as <em>you</em> know the difference. We count builds twice so we can show both "number of builds" and "percentage failed" I think: <a href="/user_uploads/4715/AGx5bz9D8KtcEEW0JL6Gcrgm/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/AGx5bz9D8KtcEEW0JL6Gcrgm/image.png" title="image.png"><img src="/user_uploads/4715/AGx5bz9D8KtcEEW0JL6Gcrgm/image.png"></a></div>



<a name="236832554"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236832554" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236832554">(Apr 30 2021 at 13:20)</a>:</h4>
<p>I think it should be possible to just use a single metric</p>



<a name="236832630"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236832630" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236832630">(Apr 30 2021 at 13:20)</a>:</h4>
<p>like <code>requests { endpoint, outcome, channel, asm_flavor, edition }</code>...</p>



<a name="236832716"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236832716" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236832716">(Apr 30 2021 at 13:21)</a>:</h4>
<p>and then for the ones that aren't relevant either put the default or an empty string</p>



<a name="236834876"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236834876" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236834876">(Apr 30 2021 at 13:36)</a>:</h4>
<p>possible, sure. Is it a good idea?</p>



<a name="236835422"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236835422" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236835422">(Apr 30 2021 at 13:40)</a>:</h4>
<p>I <em>think</em> this is the full set of labels:</p>
<div class="codehilite"><pre><span></span><code>endpoint
outcome

AssemblyFlavor
DemangleAssembly
ProcessAssembly
CompileTarget
Channel
Mode
Edition
CrateType
LibraryType
tests?
backtrace?
</code></pre></div>



<a name="236836043"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236836043" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236836043">(Apr 30 2021 at 13:44)</a>:</h4>
<blockquote>
<p>A label with an empty label value is considered equivalent to a label that does not exist.</p>
</blockquote>
<ul>
<li><a href="https://prometheus.io/docs/concepts/data_model/">source</a></li>
</ul>



<a name="236836413"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236836413" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236836413">(Apr 30 2021 at 13:47)</a>:</h4>
<p>ok dokie then; guess I'll rewrite my code again ;-)</p>



<a name="236844361"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236844361" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236844361">(Apr 30 2021 at 14:39)</a>:</h4>
<p>Seems plausible, next to try something with a missing field:</p>
<div class="codehilite"><pre><span></span><code># HELP playground_requests_total Number of requests made
# TYPE playground_requests_total counter
playground_requests_total{channel=&quot;Beta&quot;,edition=&quot;Rust2018&quot;,endpoint=&quot;Execute&quot;,mode=&quot;Debug&quot;,outcome=&quot;ErrorTimeoutSoft&quot;} 1
playground_requests_total{channel=&quot;Beta&quot;,edition=&quot;Rust2018&quot;,endpoint=&quot;Execute&quot;,mode=&quot;Debug&quot;,outcome=&quot;ErrorUserCode&quot;} 1
</code></pre></div>



<a name="236848973"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236848973" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236848973">(Apr 30 2021 at 15:05)</a>:</h4>
<p>Example:</p>
<div class="codehilite"><pre><span></span><code>playground_requests_total{backtrace=&quot;&quot;,channel=&quot;&quot;,crate_type=&quot;&quot;,edition=&quot;Rust2018&quot;,endpoint=&quot;Format&quot;,mode=&quot;&quot;,outcome=&quot;Success&quot;,tests=&quot;&quot;} 1
playground_requests_total{backtrace=&quot;false&quot;,channel=&quot;Beta&quot;,crate_type=&quot;Binary&quot;,edition=&quot;Rust2018&quot;,endpoint=&quot;Execute&quot;,mode=&quot;Debug&quot;,outcome=&quot;Success&quot;,tests=&quot;false&quot;} 1
</code></pre></div>



<a name="236854137"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236854137" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236854137">(Apr 30 2021 at 15:31)</a>:</h4>
<p>another thing you should probably consider is adding auth to the metrics endpoint</p>



<a name="236854225"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236854225" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236854225">(Apr 30 2021 at 15:32)</a>:</h4>
<p>hm, why?</p>



<a name="236854333"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236854333" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236854333">(Apr 30 2021 at 15:32)</a>:</h4>
<p>I've been adding it lately, mostly to prevent others to scrape the metrics</p>



<a name="236854457"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236854457" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236854457">(Apr 30 2021 at 15:33)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20metrics.20tracking/near/236854137">said</a>:</p>
<blockquote>
<p>another thing you should probably consider is adding auth to the metrics endpoint</p>
</blockquote>
<p>just requiring a <code>Authorization: Bearer {random_secret}</code> with the secret configured with an env var should be enough if we want to do so</p>



<a name="236854506"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236854506" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236854506">(Apr 30 2021 at 15:33)</a>:</h4>
<p>Yeah I mean, I understand the mechanics, but it seems like an unnecessary complication. I guess not opposed.</p>



<a name="236863810"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/236863810" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#236863810">(Apr 30 2021 at 16:35)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20metrics.20tracking/near/236854333">said</a>:</p>
<blockquote>
<p>to prevent others to scrape the metrics</p>
</blockquote>
<p>I missed it: why this is desirable?</p>



<a name="237707921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237707921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237707921">(May 06 2021 at 18:35)</a>:</h4>
<p>OK, merging the commit to add metrics. This will also restart the playground backend server less frequently. Previously, I always restarted it every time the cronjob ran, now only when the md5sum changes.</p>



<a name="237708942"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237708942" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237708942">(May 06 2021 at 18:42)</a>:</h4>
<p>How often was that?</p>



<a name="237710574"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237710574" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237710574">(May 06 2021 at 18:53)</a>:</h4>
<p>On my instance, every hour (at :43 past)</p>



<a name="237710797"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237710797" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237710797">(May 06 2021 at 18:55)</a>:</h4>
<p>And similar for official:</p>
<div class="codehilite"><pre><span></span><code>Description = Update the playground every hour

[Timer]
OnBootSec = 10min
OnUnitActiveSec = 1h

[Install]
WantedBy = timers.target
</code></pre></div>



<a name="237710842"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237710842" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237710842">(May 06 2021 at 18:55)</a>:</h4>
<p>ah ok</p>



<a name="237710911"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237710911" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237710911">(May 06 2021 at 18:55)</a>:</h4>
<p>Basically wanted to reduce the possibility of "lost" metrics</p>



<a name="237730732"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237730732" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237730732">(May 06 2021 at 21:13)</a>:</h4>
<p>And they are live now</p>



<a name="237746228"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237746228" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237746228">(May 06 2021 at 23:32)</a>:</h4>
<p><span class="user-mention" data-user-id="116122">@simulacrum</span> any feelings on if we should bang it into our Prometheus instance right away or do something (what?) first?</p>



<a name="237746294"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237746294" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237746294">(May 06 2021 at 23:33)</a>:</h4>
<p>I want to check-in with Pietro tomorrow morning on the upgrade so we don't race the two patches but otherwise am happy to deploy it</p>



<a name="237746786"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237746786" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237746786">(May 06 2021 at 23:39)</a>:</h4>
<p>Ah yeah, forgot that update was happening. I might set up my own instance to monitor the i32 instance too.</p>



<a name="237835121"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237835121" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237835121">(May 07 2021 at 14:50)</a>:</h4>
<p>Pretty graphs are addictive</p>
<p><a href="/user_uploads/4715/ubvJ6wGHIsT2NCSvuCMLm7c5/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/ubvJ6wGHIsT2NCSvuCMLm7c5/image.png" title="image.png"><img src="/user_uploads/4715/ubvJ6wGHIsT2NCSvuCMLm7c5/image.png"></a></div>



<a name="237836123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237836123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237836123">(May 07 2021 at 14:56)</a>:</h4>
<p>I love that about 1/2 of the requests result in the compiler saying "nah"</p>



<a name="237836151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237836151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237836151">(May 07 2021 at 14:56)</a>:</h4>
<p>(or the user program erroring in some way)</p>



<a name="237837248"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237837248" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237837248">(May 07 2021 at 15:03)</a>:</h4>
<p>uh did we configure that already with our prometheus instance?</p>



<a name="237837735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237837735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237837735">(May 07 2021 at 15:06)</a>:</h4>
<p>no</p>



<a name="237837768"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237837768" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237837768">(May 07 2021 at 15:06)</a>:</h4>
<p>Jake's running it on some other server, I think</p>



<a name="237842073"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237842073" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237842073">(May 07 2021 at 15:31)</a>:</h4>
<p>hmm, why?</p>



<a name="237842094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237842094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237842094">(May 07 2021 at 15:31)</a>:</h4>
<p>we shouldn't really have separate monitoring systems</p>



<a name="237850829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237850829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237850829">(May 07 2021 at 16:24)</a>:</h4>
<blockquote>
<p>I want to check-in with Pietro tomorrow morning on the upgrade so we don't race the two patches but otherwise am happy to deploy it</p>
</blockquote>



<a name="237850921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237850921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237850921">(May 07 2021 at 16:24)</a>:</h4>
<blockquote>
<p>I might set up my own instance to monitor the i32 instance too.</p>
</blockquote>



<a name="237850979"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237850979" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237850979">(May 07 2021 at 16:25)</a>:</h4>
<p>yeah I guess I should explicitly say - <span class="user-mention" data-user-id="121055">@Pietro Albini</span> should we be blocked on this for the upgrade? I don't know what the status there is</p>



<a name="237851004"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851004" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851004">(May 07 2021 at 16:25)</a>:</h4>
<p>So I’m also scraping the real metrics for now until they are added in the real place</p>



<a name="237851041"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851041" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851041">(May 07 2021 at 16:25)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20metrics.20tracking/near/237850979">said</a>:</p>
<blockquote>
<p>yeah I guess I should explicitly say - <span class="user-mention silent" data-user-id="121055">Pietro Albini</span> should we be blocked on this for the upgrade? I don't know what the status there is</p>
</blockquote>
<p>hmm, what upgrade? the version upgrade?</p>



<a name="237851052"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851052" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851052">(May 07 2021 at 16:25)</a>:</h4>
<p>there should be no need to block on that</p>



<a name="237851446"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851446" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851446">(May 07 2021 at 16:29)</a>:</h4>
<p>ok, so I can go ahead and enable it then?</p>



<a name="237851569"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851569" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851569">(May 07 2021 at 16:30)</a>:</h4>
<p>yep</p>



<a name="237851622"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851622" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851622">(May 07 2021 at 16:30)</a>:</h4>
<p>sounds good</p>



<a name="237851795"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237851795" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237851795">(May 07 2021 at 16:31)</a>:</h4>
<p>Ping me when it starts so I know to turn mine off, please.</p>



<a name="237852770"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237852770" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237852770">(May 07 2021 at 16:37)</a>:</h4>
<p><span class="user-mention" data-user-id="116155">@Jake Goulding</span> alright we're monitoring I think, you should be able to login and add it</p>



<a name="237869489"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237869489" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237869489">(May 07 2021 at 18:30)</a>:</h4>
<p>Looking forward to that Grafana update already <span aria-label="sweat smile" class="emoji emoji-1f605" role="img" title="sweat smile">:sweat_smile:</span></p>



<a name="237885727"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237885727" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237885727">(May 07 2021 at 20:30)</a>:</h4>
<p>Any idea why zooming in on a time range says that there are no data points? <a href="https://grafana.rust-lang.org/d/BoFKl4CGk/timeouts?orgId=1&amp;from=1620418487584&amp;to=1620419387584">https://grafana.rust-lang.org/d/BoFKl4CGk/timeouts?orgId=1&amp;from=1620418487584&amp;to=1620419387584</a></p>



<a name="237886359"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237886359" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237886359">(May 07 2021 at 20:35)</a>:</h4>
<p>it's showing data?</p>



<a name="237886674"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237886674" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237886674">(May 07 2021 at 20:37)</a>:</h4>
<p><span class="user-mention" data-user-id="116155">@Jake Goulding</span> also, could you rename the dashboards to "playground $foo"? otherwise on the recently viewed dashboards they're just "health"</p>



<a name="237889206"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237889206" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237889206">(May 07 2021 at 20:59)</a>:</h4>
<p>Then what's the point of folders...</p>



<a name="237889312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237889312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237889312">(May 07 2021 at 21:00)</a>:</h4>
<p>organization in the full list</p>



<a name="237889344"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237889344" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237889344">(May 07 2021 at 21:00)</a>:</h4>
<p>I agree it's annoying that the folder name is not displayed in the "recent dashboards"</p>



<a name="237889469"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237889469" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237889469">(May 07 2021 at 21:01)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/playground.20metrics.20tracking/near/237886359">said</a>:</p>
<blockquote>
<p>it's showing data?</p>
</blockquote>
<p><a href="/user_uploads/4715/xaMmHAuckYRyzr2v1H6MPO_n/image.png">image.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/xaMmHAuckYRyzr2v1H6MPO_n/image.png" title="image.png"><img src="/user_uploads/4715/xaMmHAuckYRyzr2v1H6MPO_n/image.png"></a></div>



<a name="237889572"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237889572" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237889572">(May 07 2021 at 21:02)</a>:</h4>
<p>oh when zooming?</p>



<a name="237912791"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237912791" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237912791">(May 08 2021 at 01:20)</a>:</h4>
<blockquote>
<p>Any idea why zooming in on a time range says that there are no data points?</p>
</blockquote>



<a name="237941130"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237941130" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237941130">(May 08 2021 at 09:18)</a>:</h4>
<p>Did you set up the metric scrape period in grafana data source settings a right? Should be largest of the scrape frequencies between all metric targets.</p>



<a name="237941246"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237941246" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237941246">(May 08 2021 at 09:20)</a>:</h4>
<p>Another thing is to use <code>rate(metric[$__rate_interval])</code> instead of just $__interval. (Same for other rate-like operations)</p>



<a name="237941478"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237941478" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237941478">(May 08 2021 at 09:24)</a>:</h4>
<p>Can't see the dashboard so can't really inspect what the exact cause is but these do come up with people making dashboards all the time.</p>



<a name="237974465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/237974465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#237974465">(May 08 2021 at 18:10)</a>:</h4>
<p>Yep, I expect that those are the likely culprits, as I ran into them when doing my local setup.</p>
<p>However, <strong>simulacrum</strong> set up the scraping, so I'm not sure what the interval is. Also <strong>Pietro Albini</strong> hasn't had an opportunity to upgrade to Grafana 7.x, so I don't think that we have access to <code>$__rate_interval</code> yet (at least it doesn't show up in the autocomplete)</p>



<a name="241983845"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/playground%20metrics%20tracking/near/241983845" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/playground.20metrics.20tracking.html#241983845">(Jun 08 2021 at 20:48)</a>:</h4>
<p>And now Grafana 8 has been released — good thing we held off the update! :-)</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>